Skip to content

chore(ingestion): report duplicate observations without psql - #850

Merged
jirhiker merged 1 commit into
stagingfrom
chore/duplicate-report-script
Aug 19, 2026
Merged

chore(ingestion): report duplicate observations without psql#850
jirhiker merged 1 commit into
stagingfrom
chore/duplicate-report-script

Conversation

@jirhiker

Copy link
Copy Markdown
Member

The SQL version of the pre-migration duplicate check needs a psql client and a
database password. A machine with only gcloud has neither, which is how this
check stalled when it was needed.

This does the same counting through the Cloud SQL connector with IAM
authentication: the credential is the operator's own gcloud login, and nothing
is stored.

uv run --group ingestion python -m \
  automated_ingestion.scripts.report_duplicate_observations \
  --instance waterdatainitiative-271000:us-west4:dataservices \
  --database ocotillo-staging

It reports, it does not resolve

Groups whose value disagrees are conflicting measurements rather than
redundant rows. Collapsing them discards a reading somebody recorded, so which
copy survives is a judgement about the data and not something a script should
make.

The failure path is most of the value

A principal without a database user gets a Postgres error that reads exactly
like a wrong password:

FATAL: password authentication failed for user "..."

so the script names the gcloud sql users create command that fixes it. That
mattered in practice — being a project owner does not imply a database login,
and the error does not say so.

Note that a user still needs USAGE on public and SELECT on
transducer_observation; creating the IAM user alone is not enough, and that
grant needs someone who already has privileges.

Context

Written while checking #847's migration was safe to run. The check has since
come back clean for staging and production, and #847 is merged — this stays
because the next constraint migration will want it, and because it documents how
to query these databases from a machine with only gcloud.

🤖 Generated with Claude Code

The SQL version of this check needs a psql client and a database password,
neither of which is available on a machine that has only gcloud. This does the
same counting through the Cloud SQL connector with IAM authentication, so the
credential is the operator's own login and nothing is stored.

It reports and does not resolve. Which copy to keep is a judgement about the
data: groups whose values disagree are conflicting measurements rather than
redundant rows, and collapsing them would discard a reading somebody recorded.

The failure path is the useful part -- a principal without a database user gets
a Postgres authentication error that reads like a bad password, so the message
names the gcloud command that creates one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Coverage

79.45% total — gate is 75%.

Coverage for the Python files changed in this PR
Name Stmts Miss Cover Missing
automated_ingestion/scripts/report_duplicate_observations.py 51 51 0% 40-150
TOTAL 51 51 0%

@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Your pull request is automatically being deployed to Dagster Cloud.

Location Status Link Updated
ocotillo-automated-ingestion View in Cloud Aug 19, 2026 at 06:24 AM (UTC)

@jirhiker
jirhiker merged commit 8ed7784 into staging Aug 19, 2026
10 checks passed
@jirhiker
jirhiker deleted the chore/duplicate-report-script branch August 19, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant